Temporal Rewards for Performance Evaluation

Today many formalisms exist for specifying complex Markov chains. In contrast, formalism for specifying the quantitative properties to analyze have remained quite primitive. In this paper a new formalism of temporal rewards that allows complex quantitative properties (including delay type measures) to be expressed in the form of a temporal reward formula. Together, an initial (discrete-time) Markov chain and the temporal reward formula implicitly define an extended Markov chain that allows the determination of the quantitative property by traditional techniques for computing long-run averages. A method to construct the extended chain is given and it is proven that this method leaves long-run averages invariant for atomic (non-temporal) rewards.